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This paper makes the case that if e-learning research and development projects are to be successfully 
adopted in real-world teaching and learning contexts, then they must effectively address accessibil- 
ity and usability issues; and that these need to be integrated throughout the project. As such, 
accessibility and usability issues need to be made explicit in project documentation, along with allo- 
cation of appropriate resources and time. We argue that accessibility and usability are intrinsically 
inter-linked. An integrated accessibility and usability evaluation methodology that we have devel- 
oped is presented and discussed. The paper draws on a series of mini-case studies from e-learning 
projects undertaken over the past 10 years at the Open University. 


Introduction 

The Open University (OU) is Europe’s largest educational establishment, delivering 
mainly distance learning courses. The OU currently has 180,000 active students, of 
which about 9900 declare a disability (~5.5%). Approximately one-half of all disabled 
students receive some form of support from the university to enable them to partici- 
pate in their studies. The OU has made extensive investment in e-learning particu- 
larly since the late 1990s. Investment continues with an ongoing multi-million pound 
virtual learning environment (VLE) programme. The OU has a commitment to 
widening access to higher education, to providing high-quality, interactive educa- 
tional materials that meet students’ needs and operating within the mission of ‘open- 
ness to all’. The OU is committed to making its online educational content and 
student services accessible to disabled students and usable by all; a considerable 
challenge given the size of the OU student population. 
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The paper draws on examples from various projects undertaken within the OU 
over the past 10 years, presented as a series of short case studies. The purpose is not 
to highlight any failings in the way accessibility and usability was managed in any 
particular project, but rather to draw lessons from a wide range of past and ongoing 
work to inform the design of future projects. These case studies are used as the basis 
of identifying a set of lessons learnt, along with recommendations to enable future e- 
learning projects to successfully integrate and embed accessibility and usability 
considerations. 


Definitions of accessibility and usability 

Usability is the extent to which a system can be used by specified users to achieve 
specified goals with effectiveness, efficiency and satisfaction in a specified context of 
use (Karat, 1997). Usability, in an e-learning context, can thus be defined as the 
effectiveness, efficiency and satisfaction with which users can achieve specified learn- 
ing (or learning related) goals in a particular environment or with a particular tool or 
learning resource. Some regard usability as synonymous with ‘ease of use’. The IMS 
Accessibility SIG defined ‘accessibility’ as the ability of the learning environment to 
adjust to the needs of all learners (IMS Global Learning Consortium, 2002). Acces- 
sibility is thus determined by the flexibility of the e-learning system or learning 
resource to meet the needs and preferences of all users. These needs and preferences 
may arise from their environment (e.g. working in a noisy environment), the tools 
they use (e.g. assistive technologies such as screen-readers, voice-recognition tools or 
alternative keyboards, etc.) or a disability in the conventional sense. 

Accessibility and usability are intrinsically linked. The lower the level of accessibil- 
ity of a resource for an individual, the less usable it will be for them. In the worst case 
they will not be able to use it at all. Conversely, improved accessibility for disabled 
users promotes usability for all. Usability should play an important role in accessibil- 
ity testing, since a resource presenting usability difficulties will generally present 
significant accessibility problems for disabled users (Sloan et al., 2002). Even sites 
with a high level of accessibility can nevertheless have usability problems that may 
prevent people with disabilities from using them efficiently. 

Most projects developing or exploring the use of technologies in education wish to 
see their approaches adopted in teaching and learning practice beyond the term of the 
project. Adequately addressing accessibility and usability in their developments 
enhances the possibility of achieving this. One important reason for this is that in 
most countries there is now anti-discrimination legislation 1 relating to disabled 
people’s access to education. The introduction of e-learning technologies should not 
put barriers in the way of disabled students accessing their learning. Further, in the 
terms of the UK legislation, reasonable adjustments should be made to meet the 
needs of disabled students in accessing the curriculum. One such reasonable adjust- 
ment is addressing accessibility and usability issues in e-learning developments. If this 
is not adequately done, then a judgment has to be made as to whether that particular 
development can in fact be deployed. 
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Accessibility and usability impact directly on the pedagogical effectiveness of e- 
learning systems or resources for all learners, but particularly for disabled learners. 
This should be reason enough for them to be addressed in all e-learning projects. It 
is invariably the case that addressing accessibility in the development phase is far 
more cost-effective than any retrospective accessibility response and is usually less 
costly and better pedagogically than the provision of an alternative but comparable 
learning experience for disabled students. The main challenge in accessibility is 
responding to the diversity of the ways different users interact with a computer envi- 
ronment. As well as being encouraged to follow established accessibility guidelines, 
developers need to be encouraged to always bear in mind that people interact with 
computers in different ways. There is globally a lack of in-depth knowledge and 
expertise in accessibility and usability available to project teams. 


Evaluation methodologies 

Evaluation is key to any research and development effort. If we do not evaluate, how 
do we know whether our developments achieved what we set out to achieve? 
Evaluation is also needed to enable iterative improvement. A range of different eval- 
uations is required during a research and development project, including technical 
and functional evaluations. However, in reviewing how best to embed accessibility 
and usability in research and development projects, this paper focuses on evaluation 
of the end-user experience. This section discusses issues of designing and integrating 
accessibility and usability evaluation methods generally, and then outlines a method- 
ology developed by the authors. Comments are made based on the experience of 
using and refining this methodology in different projects. 


Methods 

When planning an evaluation it is important that the associated aims are clearly artic- 
ulated, as these will directly impact on the selection of methods and the associated 
design of the evaluation. Different methods are likely to be appropriate at different 
stages of the project depending on the objectives of the evaluation and the status of 
the prototype being evaluated at that stage. This section discusses a range of different 
approaches that are common in evaluation work looking at accessibility and usability. 

In our work we define an expert evaluation as one undertaken by an accessibility or 
usability expert in which they make an assessment of probable issues for users in inter- 
acting with the prototype being evaluated. They will usually try and replicate a user’s 
interaction with the website or software by working through scenarios or typical tasks 
expected of the users. In assessing the accessibility of a web site or software applica- 
tion, the expert will interact with the interface using a range of assistive technologies. 
Routinely in our work that will be a screen-reader, a screen magnifier and voice- 
control software. The expert will attempt to perform all user actions without the use 
of a mouse and test the response of the software to changes in browser or operating 
system accessibility settings, such as text size and colour contrast. 
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There are automated accessibility evaluation tools available such as the desktop 
application Bobby™ (Watchfire®) 2 and the online service A-Checker. 3 These are 
designed to check web sites against the criteria of existing accessibility guidelines, 
including Section 508 of the US Rehabilitation Act of 1973 4 and the W3C’s Web 
Content Accessibility Guidelines. 5 These tools have their place in supporting expert 
evaluations but, in our view, come with a ‘handle with care’ label. Such tools evalu- 
ate technical accessibility. They can be very useful in rapidly discovering certain 
accessibility problems, but they cannot assess from the user perspective. It is 
perfectly possible for an educational web site, say, to meet all the Web Content 
Accessibility Guidelines accessibility criteria as assessed by such tools but not to 
give a disabled student access to the learning. This follows from issues relating to 
usability, learning design and how these interact with accessibility. A simple illus- 
tration of this concerns the use of alt-texts 6 and images. Such technically focused 
tools will highlight where ‘alt’ attributes on image elements are empty but do not 
assess whether any text in an ‘alt’ attribute is pedagogically meaningful. Similarly, 
a learning resource may be technically accessible but its design means a screen- 
reader user takes so long to navigate around it that they give up before achieving or 
even discovering the learning objective. This illustrates one of the central points of 
this paper that accessibility, usability and pedagogic issues are all interrelated in an 
e-learning context. 

We believe that end-user engagement is vital, and we would strongly recommend 
that projects build in the opportunity to undertake end-user evaluation as part of 
their project plan. A range of methods is available for engaging end-users in an 
evaluation of prototype software, web resources or applications. These methods are 
designed to enable the researcher to elicit the user’s experience including their 
behaviour, their perceptions and cognitive changes, their affective responses and 
their views. They are normally devised so that comparisons between the experi- 
ences of different users are possible. It has been our experience in evaluations 
undertaken that we always gain additional insights and reveal further usability and 
accessibility issues when we conduct evaluations with users, including users with 
disabilities. 

Asking direct questions of the participants is a basic way of eliciting information 
about their interaction with a prototype under evaluation, although such instruments 
need careful design. In our work we make extensive use of semi-structured interviews 
straight after an observational session; with the former considering usability and 
accessibility criteria, and the latter focusing on validating whether the established 
criteria have been met. When undertaking observational studies, some means of 
recording the students’ interaction with the object of evaluation is required together 
with their reaction to the experience. At our own institution we have a dedicated 
‘Data Capture Suite’ for this purpose. The facilities allow for the synchronous 
recording of the user actions and the software behaviour on the screen together with 
video of their facial expressions and body language and audio recording of anything 
they say. Other tools may be useful; some researchers also use keystroke recorders 
and eye-trackers, for example. 
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Making research methods more inclusive 

This section discusses a range of issues around promoting the effective engagement 
of disabled people in research including evaluation studies. As with most types of 
research, in accessibility and usability evaluations framing the right research ques- 
tions is essential to the detailed design of the method, running of the evaluation 
sessions and subsequent data analysis. Research questions focus on the following 
aspects: ‘to what extent is the learning resource, web site or software application 
accessible to disabled students?’, ‘how does this compare with usability for non- 
disabled students?’, ‘what are disabled students’ opinions on ...?’ and ‘how does this 
compare with those of non-disabled students?’ However, it is important to note that 
some care needs to be taken in phrasing of these questions. For example, the question 
‘is this web site/software accessible?’ is inappropriate because a given web site may be 
accessible to some people but not to others. Similarly there are different degrees of 
accessibility, which will also be influenced by additional tools the user might use to 
facilitate their access. 

This section gives some of the findings from our own reflection on adopting an 
integrated usability and accessibility methodology. We try to apply a philosophy of 
continued improvement to our own methodology. Disabled participants often 
needed more time to complete a task; this is important in designing a learning 
resource and how it is to be used, but is also important in the evaluation design. There 
are noticeable different levels of interaction in an evaluation task between disabled 
and non-disabled users. Disabled people are often aware of the need for accessibility 
and focus at a ‘technical’ level; that is, whether they can get to the functionality. In 
other words they look for accessibility issues. Non-disabled people typically focus at 
a ‘personal use’ level (e.g. why would I want to use this website or software?). It seems 
to be the case that non-disabled users are more reluctant to be negative in their 
comments about the web site or software under evaluation. 

There is very little literature on inclusion of disabled people in research. Accessibil- 
ity research often focuses on questions of ease of access for disabled users; however, 
they need also to take account of the accessibility of the evaluation methods they use. 
Sampling is important, but can be challenging in this area. In any research into users’ 
experiences, the more representative of the general target population the sample of 
users, the more likely the research is to reveal issues reflecting a diversity of users. 
Sampling of the population is a particular challenge in accessibility research. There is 
a wide diversity in the ways people with disabilities choose or need to interact with the 
computer. It is worth noting that it is the way an individual elects to interact with a 
computer environment that is important in accessibility research, not any medical 
classification of disability per se. Although we have undertaken studies with about 1 00 
users, more typically we work with about 10 disabled users in a study. This is not 
sufficient to cover the functional requirements of all disabled people in their interac- 
tions with the computer; however, we select users who are likely to be particularly 
challenged by the application under evaluation. We supplement user evaluations with 
expert judgements and we believe that it is better to engage with 10 disabled people 
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in an evaluation than none; even engaging with five disabled people can provide 
valuable insights. 

Different methods raise different issues when undertaking accessibility research. 
An illustration of some of these is provided here. Observational evaluations raise a 
number of issues, particularly in terms of ensuring that an appropriate environment 
is provided for the end-user and any necessary adaptations for disabled students 
should be organised in advance. For example, assistive technologies and/or adjustable 
furniture may be needed, facilitators (sign-language interpreters or note-takers) may 
be required, written instructions may be required for hearing impaired participants or 
large print for visually impaired participants. It is often necessary for the researcher 
to be in the same room as the participant for communication with hearing impaired 
and speech impaired participants. Finally, it may be necessary to factor in more time 
for tasks depending on the needs of those taking part in the study. Focus groups raise 
additional issues; for example, it can be difficult to include people with visual/hearing/ 
speech impairments, and access issues for other disability groups (e.g. those with 
dyslexia if flip-charting/post-it techniques are used) need to be thought through. 
Surveys provide a different set of issues to those involving face-to-face interaction. 
The primary one is whether alternative methods can be offered (e.g. telephone/paper/ 
online surveys) to ensure maximum accessibility. Similar questions arise with inter- 
viewing — are telephone/email/conferencing alternatives possible? With technology 
field trials, can adaptations for mobile technologies be used or are participants 
allowed to use their own technology? Newer techniques also raise interesting issues in 
terms of the possibility they offer. For example, how might a method that includes 
eye tracking be adapted to be inclusive with visually impaired participants? In some 
cases tracking mouse trails or the screen-reader focus might be a way of collecting 
comparable data, but this is not yet a method evaluated by the authors. Remote 
evaluations over the Internet using virtual screen software in conjunction with various 
Internet communication tools are becoming increasingly popular. This approach is 
being adopted by some as a way of making disabled testers more readily available to 
researchers (e.g. Usability Exchange Service 7 ). 

End-user evaluations can yield a body of rich and diverse data. Analysing this 
requires application of appropriate qualitative data analysis techniques. Accessibility 
and usability problems can be identified by end-users not being able to complete a set 
task, by unexpected behaviour of the system or the user, by direct reporting from the 
users, and so on. A more detailed discussion of qualitative data analysis techniques is 
outside the scope of this paper. However, evidently planning the analysis should be 
an integral process in devising the evaluation methodology for a given study. 

When first seeking to integrate and further develop established accessibility and 
usability methodologies in our work at the OU, the following objectives were 
established in the approach we adopted: 

1. To use, as far as possible, the same methods with disabled and non-disabled 
participants. 

2. To clearly identify usability and accessibility issues. 
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3. To produce a report in which there is no conflict between the recommendations 
for usability and accessibility improvements. 

4. To produce a report that is useful and usable by the developers in their subse- 
quent work. 

To a large extent all these have been met as our methodology has evolved. Objective 
one is often achieved by devising comparable methods rather than identical ones. By 
involving both disabled and non-disabled people in integrated studies of accessibility 
and usability, objective two has been achieved. By having accessibility and usability 
experts working together in the studies, objective three has been met. In fact, since 
adopting the integrated methodology no such conflicts have arisen. However, others 
have reported to us that in commissioning separate usability and accessibility studies 
they have received conflicting recommendations from each. Objective four has been 
achieved by producing, in the main succinct, bullet-point reports with clear 
numbered recommendations (so developers can treat them like a bug report), sugges- 
tions of techniques (so developers are not left with the question ‘so what do I do about 
it?’) and offers of clarification that promote an onward working relationship between 
the developers and the accessibility and usability experts. We are currently working 
on ways of enhancing the data collection opportunity in the precious time that we 
have the user in front of a prototype. We are seeking to reduce the dependence on 
video records to speed up data analysis and also to allow us more readily to extend 
our methodology to remote evaluations. We are also seeking to reduce non-disabled 
users’ ‘need to please’ and encourage critical review. 


Brief case studies of past and present projects 

This section describes five projects either internal to the OU or in which the OU has 
participated, and reviews how accessibility and usability were addressed in each of 
them. The authors either were directly involved in or were called upon to advise and 
evaluate prototypes in all of these projects. 


The Lyceum tool 

Lyceum is a synchronous groupware communication tool that was developed by the 
OU and subsequently used widely in course delivery. It facilities online group work- 
ing and was designed specifically for an educational context. It began life as a research 
project within the OU’s Knowledge Media Institute 8 that ran from 1995 to 1998. 
Following successful trials in a course context in 1999, the system was handed over 
to Learning and Teaching Solutions, 9 who have developed and maintained it ever 
since. Lyceum supports interactions between tutors and students and between 
students and students in a variety of capacities, such as tutoring, group work, negoti- 
ation, collaborative writing and peer-to-peer communication. It facilities voice 
conferencing, virtual rooms, shared concept mapping and collaborative white-board- 
ing. The authors have been involved in the evaluation of various applications of 
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Lyceum in different course and project contexts but were not involved in evaluation 
of the original research project prototype. Lyceum is coming to the end of its life now 
and a tendering process is underway for a replacement. 

Lyceum is a good example of an e-learning project that then gets embedded into 
teaching and learning practice. This take up of the tool was because many of the course 
teams (particularly in languages) could see the pedagogic potential for their subject 
area. This move from research to practice, however, has not been without its chal- 
lenges, particularly with respect to accessibility issues. The original Lyceum project 
did not include work on accessibility issues, as the researchers were concentrating at 
the time more on the pedagogic potential of the technology. Furthermore, the Lyceum 
development work occurred before the WCAG 1 .0 Web Content Accessibility Guide- 
lines were published, at a time when accessibility had a much lower profile. The unfor- 
tunate consequence of not considering accessibility in the initial project was that 
considerable re-engineering had to be done when it was handed over for mainstream 
use in order to meet even basic accessibility criteria. This substantially increased the 
development costs of Lyceum and delayed the release date of the software. However, 
improved accessibility has been one of the objectives of subsequent software upgrades. 


OpenMark-S for e-assessment 

OpenMark-S was an internal OU project concerned with the development of online 
formative and summative assessment. In particular, it supports the creation and deliv- 
ery of interactive questions that go beyond the constraints of simple multiple-choice 
questions. The project has made accessibility a high priority, and the second author 
was involved in conducting evaluation of prototypes with disabled students. An 
important lesson from this project was that the point when software is being upgraded 
provides a good opportunity to address any accessibility deficits in the versions. In 
this project the second author was involved as an accessibility expert from the early 
stage of the development phase of the project. A close working relationship was 
fostered between the developer and the accessibility expert with frequent contact. 
Because of this and because of the project design, frequent iterations were possible 
throughout the developments of the project. 


The DiVA project 

DiVA, the Digital Video Applications research project, ran from 2000 to 2004. 10 The 
project investigated the use of digital video within various academic contexts. This 
included using the DiVA system to locate clips from existing television programmes 
for re-use in other media, the pedagogical impact of the use of video and whether the 
DiVA system could improve access to video material for hearing impaired users. 
Central to the project was the evaluation of the use and impact of digital video in these 
contexts. The evaluation findings were incorporated into a final report delivered in 
May 2005 that made recommendations for the future of digital video within the OU. 
DiVA was a good example of a project that included research into the needs of disabled 
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students from the outset. Included in its research agenda was the objective of exploring 
how best to present transcripts, especially for people with a hearing impairment, 
alongside video material. A key lesson to be drawn is the value of formulating specific 
research questions relating to the needs of disabled students from the outset. 


The OU-VLE (Moodle) programme 

The ongoing OU-VLE Development Programme is developing a VLE 11 for the OU 
integrating a range of computer-mediated services around a courseware server based 
on the Open Source 12 (OS) course management system Moodle. 13 It consists of a 
range of specific projects focused on different aspects of developing the VLE. These 
include the Integrated Online Experience Project (portals and personalisation), ePort- 
folios, 14 eAssessment (online formative and summative computer-marked assess- 
ment), online collaboration and communications (asynchronous and synchronous 
tools such as Blogs, Wikis, web conferencing, instant messaging, etc.), mobile learner 
support and mathematical and scientific content. The system is now in use by courses 
across the OU. A second major production release is scheduled for February 2008. 
Accessibility has been a key priority, and the first author has been charged with overall 
responsibility for accessibility across all the projects within the programme. The 
second and third authors are undertaking a series of expert and end-user accessibility 
and usability evaluations of different components of the VLE as they reach usable 
prototypes. Moodle is an OS community development and the OU is investing consid- 
erable resources and effort to enhance the accessibility and usability of the system. 
There are advantages and disadvantages in addressing the accessibility agenda in an 
OS context as opposed to a propriety one; when a user detects an accessibility problem 
in an OS application, a fix will often occur more rapidly than with propriety software; 
however, the inherent nature of OS development — making software publicly available 
in early versions so the community can contribute to its further development — often 
means that products are released with accessibility poorly addressed. Key accessibility 
benefits follow from consistency of interface design and behaviour across an applica- 
tion. With a more loosely coordinated and distributed team of developers contributing 
to an OS product there is a risk that this consistency is not achieved. Where an OS 
development results in the rapid release of updated versions of the software, any 
changes in interface design and behaviour can be more problematic for some disabled 
users as it may take them longer to discover and understand such changes. 

The OU undertook a detailed review of many of the then available VLEs prior to 
selecting Moodle. It found that most addressed accessibility poorly. In selecting 
Moodle, although there were known accessibility problems, because it was an OS 
community project, the OU felt it could contribute to addressing these. The first two 
authors and a developer formed the core of a working group to draw up a Moodle 
Accessibility Specification. 15 This document gives general information to facilitate 
developers addressing accessibility and gives a prioritised list of specific issues to be 
addressed to improve to the accessibility of Moodle. It was based on a detailed expert 
evaluation of many (but not all) features in Moodle version 1.6. The document was 
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posted to the openly accessible Moodle Docs 16 site (so that the knowledge could be 
shared across the Moodle community), and developers at Moodle.com were then 
commissioned to undertake the software changes necessary to meet most of the high- 
priority issues identified in the specification. The second author has been undertaking 
reviews of these changes to confirm that the accessibility benefits sort have indeed 
been achieved. This and other accessibility and usability enhancements will, subject 
to evaluation, be incorporated into future versions of Moodle released and will thus 
benefit all Moodle users. 

The OU-VLE Development Programme represents a highly complex set of inter- 
related projects. Addressing accessibility across these has required action at both a 
strategic and project level, as well as at the fine detail of the specific developments. 
There is still much to do but the current status in terms of accessibility and usability 
issues is summarised here. Steady progress is being made on making Moodle more 
accessible, although there still are accessibility issues. Aspects of the VLE are not 
currently as usable as they might be, which is an issue for all students, but particularly 
affects students with disabilities, and usability studies have shown that some inter- 
faces are not sufficiently intuitive. There is also a lack of consistency across the differ- 
ent component web sites that have very different interaction styles. With this in mind 
we recognise that we need more student testing, on a more regular basis, in a more 
timely fashion, in order to ensure that we have a fully accessible and usable VLE. A 
key challenge has been how to integrate development work and evaluations in such as 
large-scale venture. Development work regularly over-runs its target delivery date. 
However, there is a significant lead-time (typically six to eight weeks) required to plan 
a set of end-user evaluations. This is because of the need to recruit participants in 
advance and to plan the detailed methodologies. An end-user evaluation requires a 
working prototype without too many bugs but further requires appropriate content to 
populate the prototype to design meaningful tasks for the participants to undertake. 
Delays in the development of prototypes and the release of appropriate content has 
resulted in evaluation sessions that have gained less insights than might otherwise 
have been possible. The experience of scheduling issues in earlier phases of the OU- 
VLE work has meant that we are now leaving much more time for testing in the 
forthcoming development phase. 


The EU4ALL project 

The European Unified Approach for Accessible Lifelong Learning EU4ALL 17 
project started in October 2006. This is a major, four-year project with overall 
funding of US$10 million (US$1.6 million allocated to the OU). The project 
addresses systemic issues in providing access for disabled learners to Lifelong 
Learning. 18 It sets forward the concept of Accessible Lifelong Learning uniting 
three key strategies: 

• Technology that mediates lifelong learning does so in an accessible way. 

• Technology is used to bring specialist support services to disabled learners. 
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• Support services and a technical infrastructure are provided to enable staff at 
educational institutions to more readily offer their teaching and services in a way 
that is accessible to disabled learners. 

The aim of EU4ALL is to improve the efficiency and efficacy of implementing these 
strategies by developing an open service architecture. To achieve a wide impact, the 
approach taken is not to develop a single EU4ALL system but a standards-based 
framework that facilitates the integration of the approach with a wide range of e- 
learning systems. This will be validated by integration with two OS VLEs: dotLRN 19 
and Moodle. EU4ALL is in the early stages so comments here are confined to issues 
associated with project design. Our prior experience of projects of this kind suggests 
usability and accessibility should be integrated across the different phases of the 
project (requirements capture, specification, design, development and validation). In 
such a large-scale project this means integrating this work in the planning stage 
between many of the subprojects. This is a challenge in project design, and subse- 
quently project management. The EU4ALL project is addressing many issues of 
accessibility at a systems level rather than just at the interface level. This includes 
personalisation to the needs of different users with disabilities and the automated 
serving of alternative resources where required. Designing end-user evaluations that 
can contribute early to the iterative development of these is challenging. This follows 
from the fact that many technical issues need to be addressed to make these systems 
work before a meaningful user experience can be created for evaluation. 


Recommendations 

This section gives a set of recommendations for future projects based on our analysis 
of the experience in mini-case studies described in this paper. We have argued that 
evaluation has an essential role in embedding of accessibility and usability into a 
project plan. Specific numbered recommendations are listed here under different 
areas for consideration. 


Project proposals 

1 . The rationale for usability and accessibility should be clearly stated in the proposal. 

2. There should be appropriate allocation of how the usability and accessibility work 
will be resourced in the project. 

3. Usability and accessibility should be reflected in the project research questions, 
which should include something that involves an investigation of usability for all 
and accessibility for disabled students. 


Project design, development and evaluation 

4. Accessibility and usability criteria need to be built into the specification. 

5. Accessibility and usability experts need to be brought in the earliest possible stage. 
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6. If a use-case-based approach for arriving at a specification is adopted, then use- 
cases should be developed that include disabled people as ‘actors’. 

7. If integrating with third-party software, its accessibility and usability needs to be 
reviewed. 

8. The project should address the fact that accessibility and usability are system 
design issues, not just interface design issues. 

9. An integrated evaluation plan should be adopted that includes accessibility, 
usability and educational evaluation. 

10. Outline methodologies for the planned evaluations need to be produced early — 
this will effect the detailed planning of the developers work. 

1 1 . Accessibility issues associated with any data collection methods used need to be 
fully considered and accommodations devised to meet the meets of disabled 
people as necessary. 


Iterative development 

Iterative design is essential to any software development project; however, making 
this effective requires a close integration of the work carried out by the developers and 
the evaluators involved in the project. Therefore, the following issues need to be 
addressed early in any project: 

12. Where in the project plan are expert and end-user evaluations most appropriate? 

13. Sufficient time and development resource allocation after an evaluation is 
required to respond to any recommendations that arise from the evaluation. 

14. To be effective end-user evaluations, the prototypes under investigation needs to 
be populated with appropriate content and activates for the users to interact with 
in a realistic way. 

15. End-user evaluations require a reasonable lead time to facilitate the recruitment 
of participants and the preparation of the detail of the evaluation methods, which 
will be dependent on the content and activities. 

16. Development work almost inevitably takes longer than anticipated. Contingency 
plans need to be in place to deal with this. 


Successfully project managing for usability and accessibility 

How successfully issues relating to the end-user experience are addressed in a project 
depends on how accessibility and usability are managed within the project. This is 
because accessibility and usability issues transcend many areas of a project. It is 
appropriate to liken responsibility for accessibility in a project to health and safety at 
work policies, where everyone has responsibility for health and safety but specific 
duties and areas of responsibility are delegated to named individuals. Similarly, 
everyone in an e-learning research and development project team has to pay due 
regard to accessibility and usability issues. However, experience in past projects has 
shown that it is often advantageous to have a named member of the project team with 
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overarching responsibility for accessibility and usability. This person should have the 
permission of the project team to repeatedly ask ‘what are the accessibility and 
usability implications for this?’ 

The problem of integrating evaluation work with development schedules that often 
slip has been discussed above. Probably the only way to address this is by detailed 
initial planning and periodic progress and planning reviews. Close communications 
between development and evaluation teams is important so that early notice is given 
if slippages are anticipated and plans can be adjusted accordingly. There is inevitably 
a tension between developers and evaluators of software. The former perceive the 
latter of criticising work in which they have usually invested significant amounts of 
effort and professional expertise. The evaluators see the developers as reluctant to 
respond to their recommendations. Better relations and a better outcome for the 
project are promoted if an integrated team approach is fostered; with developers and 
evaluators seeing each other as partners with a common goal, calling on each others’ 
areas of expertise as necessary. To achieve this, evaluators should be involved 
throughout the project, participating in the specification work from the outset and 
being available to developers at any point. For example, a developer may ask a 
question of the form: ‘I was thinking of implementing <afunction> this way, what do 
you think will be the accessibility and usability implications of that?’ Conversely, an 
evaluator might ask: ‘I have been looking at how we have implemented < afunction>, 
can you advise me whether it can be adapted to incorporate <an approach> that I 
think will promote accessibility?’. 


Concluding comments 

In summary, a few of the more important points from the discussion of this paper are 
highlighted here. We argue that accessibility, usability and pedagogic issues are all 
interrelated in an e-learning context. Accessibility and usability issues need to be 
addressed throughout a project’s lifecycle to ensure its developments are subse- 
quently adopted in educational delivery. Evaluations are key to ensuring accessibility 
and usability issues have been addressed in a project’s developments. Valuable 
insights can be gained when conducting evaluations with a range of users leading to 
overall improvements in the system being developed. We believe that integrating 
accessibility and usability evaluations yields distinct benefits over treating them as 
separate areas for study. We have described how care is needed when drawing up 
evaluation methodologies, to ensure that they are inclusive and consistent for both 
disabled people and non-disabled people acting as participants. However, we have 
also discussed how accessibility and usability raise fundamental issues for how 
projects are organised and run. 

We believe that enhancing the end-user experience of an e-learning system or 
resource, by comprehensively addressing accessibility and usability, impacts posi- 
tively on the effectiveness of the learning, and hope that we have illustrated why this 
should be a key reason for giving them due consideration, effort and resources in any 
e-learning project. 
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Notes 

1 . For example, in the United Kingdom this is the Disability Discrimination Act 1995 (amended). 
Available online at: http://www.opsi.gov.uk/acts/actsl995/1995050.htm (accessed 27 May 
2007). 

2. Watchfire“ Bobby™, see: http://www.watchfire.com/products/webxm/bobby.aspx (accessed 
24 May 2007). 

3. A-Checker can be accessed online at: http://checker.atrc.utoronto.ca/index.html (accessed 24 
May 2007). 

4. As amended 29 U.S.C. § 794 (d) 1998, the associated Section 508 standards establish a mini- 
mum level of accessibility. See: http://www.section508.gov/ (accessed 24 May 2007). 

5. W3C Web Accessibility Initiative’s Web Accessibility Content Guidelines 1.0, see: http:// 
www.w3.org/TR/WAI-WEBCONTENT/ (accessed 24 May 2000). Web Accessibility Content 
Guidelines 2.0 anticipated, working draft available online at: http://www.w3.org/TR/WCAG20/ 
(accessed 24 May 2007). 

6. Alt-text in HTML refers to the attribute of an image element that enables the association of a 
piece of text with an image that describes that image. Some assistive technology, such as screen- 
readers, can be set to present to the user the alt-text whenever they encounter an image. By 
extension, the term alt-text is used to describe similar approaches in contexts other than 
HTML. 

7. See: http://www.usabilityexchange.com/index.php (accessed 24 May 2007). 

8. The Open University’s Knowledge Media Institute (KMi), see: http://kmi.open.ac.uk/ 
(accessed 25 May 2007). 

9. Learning and Teaching Solutions is the media production centre of the OU. Among its many 
roles it is responsible for the set-up, configuration and delivery of online learning and teaching 
services to students and staff. 

10. DiVA web site: http://library.open.ac.uk/waltonhall/diva/about.html (accessed 22 May 2007). 

1 1 . VLE is a term that applies to an integration of web-based services that facilitate the online 
interaction between an educational institution and its students, including the mediation of 
teaching and learning. 

12. OS is a community-based software development method that harnesses distributed peer review 
and open working. OS software is normally made available to the end user at low or zero cost. 

13. See: http://moodle.org/ (accessed 26 May 2007). 

14. An ePortfolio system can be defined as one supporting reflective and self-directed learning by 
enabling the collection of evidence of learning. A learner may draw upon these to identify and 
present his/her learning and achievements. 

15. Moodle Accessibility Specification, available online at: http://docs.moodle.org/en/ 

Moodle_Accessibility_Specification (accessed 26 May 2006). 

16. See: http://docs.moodle.org/ (accessed 26 May 2007). 

17. EU 1ST elnclusion funded project number 034778. See: http://www.eu4all-project.eu/ 
(accessed 26 May 2007). 

18. Lifelong Learning is the concept that all need to learn throughout their lives. It is argued that 
it is key in knowledge based economies. It encompasses the whole range of learning: formal 
and informal, workplace based, and the skills and knowledge people acquire in day-to-day 
experiences. 

19. See: http://dotlrn.org/ (accessed 26 May 2007). 
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